Testing hemifield independence for divided attention in visual object tasks.

In this study, we asked to what degree hemifields contribute to divided attention effects observed in tasks with object-based judgments. If object recognition processes in the two hemifields were fully independent, then placing stimuli in separate hemifields would eliminate divided attention effects; in the alternative extreme, if object recognition processes in the two hemifields were fully integrated, then placing stimuli in separate hemifields would not modulate divided attention effects. Using a dual-task paradigm, we compared performance in a semantic categorization task for relevant stimuli arranged in the same hemifield to performance for relevant stimuli arranged in separate left and right hemifields. In two experiments, there was a reliable decrease in divided attention effects when stimuli were shown in separate hemifields compared to the same hemifield. However, the effect of divided attention was not eliminated. These results reject both the independent and integrated hypotheses, and instead support a third alternative - that object recognition processes in the two hemifields are partially dependent. More specifically, the magnitude of modulation by hemifields was closer to the prediction of the integrated hypothesis, suggesting that for dual tasks with objects, dependent processing is mostly shared across the visual field.

PySilSub: An open-source Python toolbox for implementing the method of silent substitution in vision and nonvisual photoreception research.

The normal human retina contains several classes of photosensitive cell-rods for low-light vision, three cone classes for daylight vision, and intrinsically photosensitive retinal ganglion cells (ipRGCs) expressing melanopsin for non-image-forming functions, including pupil control, melatonin suppression, and circadian photoentrainment. The spectral sensitivities of the photoreceptors overlap significantly, which means that most lights will stimulate all photoreceptors to varying degrees. The method of silent substitution is a powerful tool for stimulating individual photoreceptor classes selectively and has found much use in research and clinical settings. The main hardware requirement for silent substitution is a spectrally calibrated light stimulation system with at least as many primaries as there are photoreceptors under consideration. Device settings that will produce lights to selectively stimulate the photoreceptor(s) of interest can be found using a variety of analytic and algorithmic approaches. Here we present PySilSub (https://github.com/PySilentSubstitution/pysilsub), a novel Python package for silent substitution featuring flexible support for individual colorimetric observer models (including human and mouse observers), multiprimary stimulation devices, and solving silent substitution problems with linear algebra and constrained numerical optimization. The toolbox is registered with the Python Package Index and includes example data sets from various multiprimary systems. We hope that PySilSub will facilitate the application of silent substitution in research and clinical settings.

Characterizing amblyopic perception under non-rivalrous viewing conditions.

Current assessments of interocular interactions in amblyopia often use rivalrous stimuli, with conflicting stimuli in each eye, which does not reflect vision under typical circumstances. Here we measure interocular interactions in observers with amblyopia, strabismus with equal vision, and controls using a non-rivalrous stimulus. Observers used a joystick to continuously report the perceived binocular contrast of dichoptic grating stimuli, identical except that the stimulus was contrast-modulated independently in each eye over time. Consistent with previous studies, a model predicting the time-course of perceived contrast found increased amblyopic eye attenuation, and reduced contrast normalization of the fellow eye by the amblyopic eye, in amblyopic participants compared to controls. However, these suppressive interocular effects were weaker than those found in previous studies, suggesting that rivalrous stimuli may overestimate the effects of amblyopia on interocular interactions during naturalistic viewing conditions.

Pulse trains to percepts: A virtual patient describing the perceptual effects of human visual cortical stimulation.

The field of cortical sight restoration prostheses is making rapid progress with three clinical trials of visual cortical prostheses underway. However, as yet, we have only limited insight into the perceptual experiences produced by these implants. Here we describe a computational model or 'virtual patient', based on the neurophysiological architecture of V1, which successfully predicts the perceptual experience of participants across a wide range of previously published cortical stimulation studies describing the location, size, brightness and spatiotemporal shape of electrically induced percepts in humans. Our simulations suggest that, in the foreseeable future the perceptual quality of cortical prosthetic devices is likely to be limited by the neurophysiological organization of visual cortex, rather than engineering constraints.

Evidence from partially valid cueing that words are processed serially.

There has been a longstanding debate about whether lexical and semantic processing of words is serial or parallel. We addressed this debate using partially valid cueing, where one of two words is cued. The cue was valid on 80% and invalid on the other 20% of the trials. The task was semantic categorization, and performance was measured by accuracy. The new feature was to limit attentional switching using a postmask of consonants that closely followed the presentation of words. We found a large effect of cueing and, most importantly, performance for the uncued word was at chance. This chance performance was consistent with serial processing, but not with typical parallel processing. This result adds to the evidence from other recent studies that the lexical and semantic processing of words is serial.

Factors affecting two-point discrimination in Argus II patients.

Two of the main obstacles to the development of epiretinal prosthesis technology are electrodes that require current amplitudes above safety limits to reliably elicit percepts, and a failure to consistently elicit pattern vision. Here, we explored the causes of high current amplitude thresholds and poor spatial resolution within the Argus II epiretinal implant. We measured current amplitude thresholds and two-point discrimination (the ability to determine whether one or two electrodes had been stimulated) in 3 blind participants implanted with Argus II devices. Our data and simulations show that axonal stimulation, lift and retinal damage all play a role in reducing performance in the Argus 2, by either limiting sensitivity and/or reducing spatial resolution. Understanding the relative role of these various factors will be critical for developing and surgically implanting devices that can successfully subserve pattern vision.

Learning to see again: Perceptual learning of simulated abnormal on- off-cell population responses in sighted individuals.

Many forms of artificial sight recovery, such as electronic implants and optogenetic proteins, generally cause simultaneous, rather than complementary firing of on- and off-center retinal cells. Here, using virtual patients-sighted individuals viewing distorted input-we examine whether plasticity might compensate for abnormal neuronal population responses. Five participants were dichoptically presented with a combination of original and contrast-reversed images. Each image (I) and its contrast-reverse (I') was filtered using a radial checkerboard (F) in Fourier space and its inverse (F'). [I * F'] + [I' * F] was presented to one eye, and [I * F] + [I' * F'] was presented to the other, such that regions of the image that produced on-center responses in one eye produced off-center responses in the other eye, and vice versa. Participants continuously improved in a naturalistic object discrimination task over 20 one-hour sessions. Pre-training and post-training tests suggest that performance improvements were due to two learning processes: learning to recognize objects with reduced visual information and learning to suppress contrast-reversed image information in a non-eye-selective manner. These results suggest that, with training, it may be possible to adapt to the unnatural on- and off-cell population responses produced by electronic and optogenetic sight recovery technologies.

A major role for retrieval and/or comparison in the set-size effects of change detection.

Set-size effects in change detection have been attributed to capacity limits in a variety of processes, including perception, memory encoding, memory storage, memory retrieval, comparison, and decision. In this study, we investigated the locus of the effect of increasing set size from 1 to 2. The task was to detect a 90 degree change in the orientation of 1 or 2 briefly presented Gabor patterns in noise. To measure purely attentional effects and not another phenomena, such as crowding, a precue was used to manipulate relevant set size while keeping the display constant. The locus of the capacity limit was determined by varying when observers were cued to a single relevant stimulus. To begin, we measured the baseline set-size effect for change detection. Next, a dual-task procedure and a 100% valid postcue was added to test for an effect of decision: This modification did not reliably change the set-size effects. In the critical experiments, a 100% valid cue was provided during the retention interval between displays, or only one stimulus was presented in the second display (local recognition). For both of these conditions, there was only a relatively small set-size effect. These results are consistent with the bulk of capacity limits being in memory retrieval or comparison and not in perception, memory encoding, or memory storage.

Is there a serial bottleneck in visual object recognition?

Divided attention has little effect for simple tasks, such as luminance detection, but it has large effects for complex tasks, such as semantic categorization of masked words. Here, we asked whether the semantic categorization of visual objects shows divided attention effects as large as those observed for words, or as small as those observed for simple feature judgments. Using a dual-task paradigm with nameable object stimuli, performance was compared with the predictions of serial and parallel models. At the extreme, parallel processes with unlimited capacity predict no effect of divided attention; alternatively, an all-or-none serial process makes two predictions: a large divided attention effect (lower accuracy for dual-task trials, compared to single-task trials) and a negative response correlation in dual-task trials (a given response is more likely to be incorrect when the response about the other stimulus is correct). These predictions were tested in two experiments examining object judgments. In both experiments, there was a large divided attention effect and a small negative correlation in responses. The magnitude of these effects was larger than for simple features, but smaller than for words. These effects were consistent with serial models, and rule out some but not all parallel models. More broadly, the results help establish one of the first examples of likely serial processing in perception.

Endogenous cueing effects for detection can be accounted for by a decision model of selective attention.

Spatial cues help participants detect a visual target when it appears at the cued location. One hypothesis for this cueing effect, called selective perception, is that cueing a location enhances perceptual encoding at that location. Another hypothesis, called selective decision, is that the cue has no effect on perception, but instead provides prior information that facilitates decision-making. We distinguished these hypotheses by comparing a simultaneous display with two spatial locations to sequential displays with two temporal intervals. The simultaneous condition had a partially valid spatial cue, and the sequential condition had a partially valid temporal cue. Selective perception predicts no cueing effect for sequential displays given there is enough time to switch attention. In contrast, selective decision predicts cueing effects for sequential displays regardless of time. We used endogenous cueing of a detection-like coarse orientation discrimination task with clear displays (no external noise or postmasks). Results showed cueing effects for the sequential condition, supporting a decision account of selective attention for endogenous cueing of detection-like tasks.

Visual word recognition: Evidence for a serial bottleneck in lexical access.

Reading is a demanding task, constrained by inherent processing capacity limits. Do those capacity limits allow for multiple words to be recognized in parallel? In a recent study, we measured semantic categorization accuracy for nouns presented in pairs. The words were replaced by post-masks after an interval that was set to each subject's threshold, such that with focused attention they could categorize one word with ~80% accuracy. When subjects tried to divide attention between both words, their accuracy was so impaired that it supported a serial processing model: on each trial, subjects could categorize one word but had to guess about the other. In the experiments reported here, we investigated how our previous result generalizes across two tasks that require lexical access but vary in the depth of semantic processing (semantic categorization and lexical decision), and across different masking stimuli, word lengths, lexical frequencies and visual field positions. In all cases, the serial processing model was supported by two effects: (1) a sufficiently large accuracy deficit with divided compared to focused attention; and (2) a trial-by-trial stimulus processing tradeoff, meaning that the response to one word was more likely to be correct if the response to the other was incorrect. However, when the task was to detect colored letters, neither of those effects occurred, even though the post-masks limited accuracy in the same way. Altogether, the results are consistent with the hypothesis that visual processing of words is parallel but lexical access is serial.

The Importance of Considering Model Choices When Interpreting Results in Computational Neuroimaging.

Model-based analyses open exciting opportunities for understanding neural information processing. In a commentary published in eNeuro, Gardner and Liu (2019) discuss the role of model specification in interpreting results derived from complex models of neural data. As a case study, they suggest that one such analysis, the inverted encoding model (IEM), should not be used to assay properties of "stimulus representations" because the ability to apply linear transformations at various stages of the analysis procedure renders results "arbitrary." Here, we argue that the specification of all models is arbitrary to the extent that an experimenter makes choices based on current knowledge of the model system. However, the results derived from any given model, such as the reconstructed channel response profiles obtained from an IEM analysis, are uniquely defined and are arbitrary only in the sense that changes in the model can predictably change results. IEM-based channel response profiles should therefore not be considered arbitrary when the model is clearly specified and guided by our best understanding of neural population representations in the brain regions being analyzed. Intuitions derived from this case study are important to consider when interpreting results from all model-based analyses, which are similarly contingent upon the specification of the models used.

The link between reading ability and visual spatial attention across development.

Interacting with a cluttered and dynamic environment requires making decisions about visual information at relevant locations while ignoring irrelevant locations. Typical adults can do this with covert spatial attention: prioritizing particular visual field locations even without moving the eyes. Deficits of covert spatial attention have been implicated in developmental dyslexia, a specific reading disability. Previous studies of children with dyslexia, however, have been complicated by group differences in overall task ability that are difficult to distinguish from selective spatial attention. Here, we used a single-fixation visual search task to estimate orientation discrimination thresholds with and without an informative spatial cue in a large sample (N = 123) of people ranging in age from 5 to 70 years and with a wide range of reading abilities. We assessed the efficiency of attentional selection via the cueing effect: the difference in log thresholds with and without the spatial cue. Across our whole sample, both absolute thresholds and the cueing effect gradually improved throughout childhood and adolescence. Compared to typical readers, individuals with dyslexia had higher thresholds (worse orientation discrimination) as well as smaller cueing effects (weaker attentional selection). Those differences in dyslexia were especially pronounced prior to age 20, when basic visual function is still maturing. Thus, in line with previous theories, literacy skills are associated with the development of selective spatial attention.

A model of ganglion axon pathways accounts for percepts elicited by retinal implants.

Degenerative retinal diseases such as retinitis pigmentosa and macular degeneration cause irreversible vision loss in more than 10 million people worldwide. Retinal prostheses, now implanted in over 250 patients worldwide, electrically stimulate surviving cells in order to evoke neuronal responses that are interpreted by the brain as visual percepts ('phosphenes'). However, instead of seeing focal spots of light, current implant users perceive highly distorted phosphenes that vary in shape both across subjects and electrodes. We characterized these distortions by asking users of the Argus retinal prosthesis system (Second Sight Medical Products Inc.) to draw electrically elicited percepts on a touchscreen. Using ophthalmic fundus imaging and computational modeling, we show that elicited percepts can be accurately predicted by the topographic organization of optic nerve fiber bundles in each subject's retina, successfully replicating visual percepts ranging from 'blobs' to oriented 'streaks' and 'wedges' depending on the retinal location of the stimulating electrode. This provides the first evidence that activation of passing axon fibers accounts for the rich repertoire of phosphene shape commonly reported in psychophysical experiments, which can severely distort the quality of the generated visual experience. Overall our findings argue for more detailed modeling of biological detail across neural engineering applications.

Parallel spatial channels converge at a bottleneck in anterior word-selective cortex.

In most environments, the visual system is confronted with many relevant objects simultaneously. That is especially true during reading. However, behavioral data demonstrate that a serial bottleneck prevents recognition of more than one word at a time. We used fMRI to investigate how parallel spatial channels of visual processing converge into a serial bottleneck for word recognition. Participants viewed pairs of words presented simultaneously. We found that retinotopic cortex processed the two words in parallel spatial channels, one in each contralateral hemisphere. Responses were higher for attended than for ignored words but were not reduced when attention was divided. We then analyzed two word-selective regions along the occipitotemporal sulcus (OTS) of both hemispheres (subregions of the visual word form area, VWFA). Unlike retinotopic regions, each word-selective region responded to words on both sides of fixation. Nonetheless, a single region in the left hemisphere (posterior OTS) contained spatial channels for both hemifields that were independently modulated by selective attention. Thus, the left posterior VWFA supports parallel processing of multiple words. In contrast, activity in a more anterior word-selective region in the left hemisphere (mid OTS) was consistent with a single channel, showing (i) limited spatial selectivity, (ii) no effect of spatial attention on mean response amplitudes, and (iii) sensitivity to lexical properties of only one attended word. Therefore, the visual system can process two words in parallel up to a late stage in the ventral stream. The transition to a single channel is consistent with the observed bottleneck in behavior.

Model-Based Recommendations for Optimal Surgical Placement of Epiretinal Implants.

A major limitation of current electronic retinal implants is that in addition to stimulating the intended retinal ganglion cells, they also stimulate passing axon fibers, producing perceptual 'streaks' that limit the quality of the generated visual experience. Recent evidence suggests a dependence between the shape of the elicited visual percept and the retinal location of the stimulating electrode. However, this knowledge has yet to be incorporated into the surgical placement of retinal implants. Here we systematically explored the space of possible implant configurations to make recommendations for optimal intraocular positioning of the electrode array. Using a psychophysically validated computational model, we demonstrate that better implant placement has the potential to reduce the spatial extent of axonal activation in existing implant users by up to ~55%. Importantly, the best implant location, as inferred from a population of simulated virtual patients, is both surgically feasible and is relatively stable across individuals. This study is a first step towards the use of computer simulations in patient-specific planning of retinal implant surgery.

Evidence of Serial Processing in Visual Word Recognition.

To test the limits of parallel processing in vision, we investigated whether people can recognize two words at once. Participants viewed brief, masked pairs of words and were instructed in advance to judge both of the words (dual-task condition) or just one of the words (single-task condition). For judgments of semantic category, the dual-task deficit was so large that it supported all-or-none serial processing: Participants could recognize only one word and had to guess about the other. Moreover, participants were more likely to be correct about one word if they were incorrect about the other, which also supports a serial-processing model. In contrast, judgments of text color with identical stimuli were consistent with unlimited-capacity parallel processing. Thus, under these conditions, serial processing is necessary to judge the meaning of words but not their physical features. Understanding the implications of this result for natural reading will require further investigation.

Reconstructing Tone Sequences from Functional Magnetic Resonance Imaging Blood-Oxygen Level Dependent Responses within Human Primary Auditory Cortex.

Here we show that, using functional magnetic resonance imaging (fMRI) blood-oxygen level dependent (BOLD) responses in human primary auditory cortex, it is possible to reconstruct the sequence of tones that a person has been listening to over time. First, we characterized the tonotopic organization of each subject's auditory cortex by measuring auditory responses to randomized pure tone stimuli and modeling the frequency tuning of each fMRI voxel as a Gaussian in log frequency space. Then, we tested our model by examining its ability to work in reverse. Auditory responses were re-collected in the same subjects, except this time they listened to sequences of frequencies taken from simple songs (e.g., "Somewhere Over the Rainbow"). By finding the frequency that minimized the difference between the model's prediction of BOLD responses and actual BOLD responses, we were able to reconstruct tone sequences, with mean frequency estimation errors of half an octave or less, and little evidence of systematic biases.

Evidence for unlimited capacity processing of simple features in visual cortex.

Performance in many visual tasks is impaired when observers attempt to divide spatial attention across multiple visual field locations. Correspondingly, neuronal response magnitudes in visual cortex are often reduced during divided compared with focused spatial attention. This suggests that early visual cortex is the site of capacity limits, where finite processing resources must be divided among attended stimuli. However, behavioral research demonstrates that not all visual tasks suffer such capacity limits: The costs of divided attention are minimal when the task and stimulus are simple, such as when searching for a target defined by orientation or contrast. To date, however, every neuroimaging study of divided attention has used more complex tasks and found large reductions in response magnitude. We bridged that gap by using functional magnetic resonance imaging to measure responses in the human visual cortex during simple feature detection. The first experiment used a visual search task: Observers detected a low-contrast Gabor patch within one or four potentially relevant locations. The second experiment used a dual-task design, in which observers made independent judgments of Gabor presence in patches of dynamic noise at two locations. In both experiments, blood-oxygen level-dependent (BOLD) signals in the retinotopic cortex were significantly lower for ignored than attended stimuli. However, when observers divided attention between multiple stimuli, BOLD signals were not reliably reduced and behavioral performance was unimpaired. These results suggest that processing of simple features in early visual cortex has unlimited capacity.